Adaptive filter for low-light noise reduction

ABSTRACT

In general, in one embodiment, low-light noise is removed from an image by separately filtering luma and chroma components of the image, by adaptively filtering the image based at least in part on a Gaussian distribution of the image, and/or by dividing the image into separate regions and filtering each region separately.

TECHNICAL FIELD

Embodiments of the invention generally relate to video signalprocessing, and in particular to processing video signals to removeartifacts caused by low-light noise.

BACKGROUND

Low-light images are especially susceptible to corruption from noisecaused by light-detecting sensors (i.e., low-light artifacts). Forexample, a video or still camera may capture undesirable grains ordiscolorations in low-light conditions. This noise may lead touncorrelated pixels and, as a result, reduced compression efficiency forvideo coding algorithms (e.g., MPEG4 and H.264). Many applications, suchas security cameras, capture low-light images and require a large amountof storage space for retaining those images, and any decrease in therequired storage space may lead to a more cost-effective application, anincrease in the number of images or frames of video stored, or reducednetwork traffic for transporting the images. Thus, efforts have beenmade to detect and eliminate low-light noise.

Previous efforts (such as transform-domain methods, DCT, wavelet, orother statistical methods), however, suffer from drawbacks. Thesemethods are computationally intensive and require a significant amountof computing resources, which may not be available on low-power,portable, or other devices. Furthermore, these methods are notadjustable based on available resources or the complexity of the sourceimage, further wasting resources on simple images or during high-loadconditions in which the additional resources may not be necessary oravailable.

SUMMARY

In general, various aspects of the systems and methods described hereinuse a Gaussian distribution and correlation technique to removeuncorrelated low-light noise from images taken from video or stillcameras. The images may be split into luma and chroma components andfiltered separately. Different filters may be used depending on thecomplexity of the images and the resources available. The filters mayadapt to variations in the image by using edge-detection and dilationfilters, thereby preserving high-frequency details at feature edges.Furthermore, the image may be divided into a plurality of sections,filtered separately, and re-combined.

In general, in one aspect, an adaptive filter for filtering noise from alow-light image includes a morphology filter and a comparative filter.The morphology filter divides the image into a non-edge region and adilated edge region by dilating an edge detected in the image. Thecomparative filter compares a pixel in the non-edge region to a regionsurrounding the pixel and optionally replaces the pixel, based on aresult of the comparison, with a new pixel derived at least in part fromthe region surrounding the pixel.

In various embodiments, a difference filter detects the edge in theimage and/or the morphology filter comprises a dilation filter. Thedilation filter may be a 3×4 dilation filter, and the region surroundingthe pixel may correspond to a 3×3 pixel region. The adaptive filter maycompare a difference between the pixel and a mean of the regionsurrounding the pixel to a variance of the region surrounding the pixel.The variance may be a variance of a Gaussian distribution of the regionsurrounding the pixel; a circuit may be included to compute the varianceof the Gaussian distribution of the region surrounding the pixel. Themean may be a mean of a Gaussian distribution of the region surroundingthe pixel; a circuit may be included to compute the mean of the Gaussiandistribution of the region surrounding the pixel.

The comparative filter may derive the new pixel based at least in parton a median value of the region surrounding the pixel and/or a low-passfilter value of the region surrounding the pixel. The comparative filtermay replace a second pixel adjacent to the pixel with the new pixel; thepixel may be an even pixel and the second pixel may be an odd pixel.Alternatively, the comparative filter may preserve the pixel.

In general, in another aspect, a method for adaptively filtering noisefrom a low-light image includes dilating an edge detected in an image todivide the image into a non-edge region and a dilated edge region. Apixel in the non-edge region is compared to a region surrounding thepixel. The pixel is optionally replaced, based on a result of thecomparison, with a new pixel derived at least in part from the regionsurrounding the pixel.

In various embodiments, the edge may be detected in the image. Comparingthe pixel to the region surrounding the pixel may include comparing adifference between the pixel and a variance of the region surroundingthe pixel to a mean of the region surrounding the pixel. The variancemay be a variance of a Gaussian distribution of the region surroundingthe pixel, and the mean may be a mean of a Gaussian distribution of theregion surrounding the pixel. The new pixel may be derived based atleast in part on a median value of the region surrounding the pixeland/or a low-pass filter value of the region surrounding the pixel. Asecond pixel, adjacent to the pixel, may be replaced with the new pixel;the pixel may be an even pixel and the second pixel may be an odd pixel.Alternatively, the comparative filter may preserve the pixel.

These and other objects, along with advantages and features of thepresent invention herein disclosed, will become more apparent throughreference to the following description, the accompanying drawings, andthe claims. Furthermore, it is to be understood that the features of thevarious embodiments described herein are not mutually exclusive and mayexist in various combinations and permutations.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like reference characters generally refer to the sameparts throughout the different views. In the following description,various embodiments of the present invention are described withreference to the following drawings, in which:

FIG. 1 is a block diagram of a system for removing noise from alow-light image in accordance with an embodiment of the invention;

FIG. 2 is a flowchart illustrating a method for removing noise from alow-light image in accordance with an embodiment of the invention;

FIG. 3 is a block diagram of an adaptive filter in accordance with anembodiment of the invention;

FIG. 4 is an example of a low-light image component in accordance withan embodiment of the invention;

FIG. 5 is a flowchart illustrating a method for adaptively filteringnoise from a low-light image in accordance with an embodiment of theinvention;

FIG. 6 is a block diagram of a system for dividing an image to removelow-light noise therefrom in accordance with an embodiment of theinvention; and

FIG. 7 is a flowchart illustrating a method for dividing an image toremove low-light noise therefrom in accordance with an embodiment of theinvention.

DETAILED DESCRIPTION

FIG. 1 illustrates a system 100 for removing noise from a low-lightimage. As one of skill in the art will understand, a source image 102may be separated into a brightness component 104 and a color component106. The brightness component 104 may also be known as a Y or lumacomponent; the color component 106 may also be known as a UV or chromacomponent. In one embodiment, the brightness component 104 and colorcomponent 106 are filtered separately using different filters. Once thebrightness component 104 and color component 106 are filtered, they maybe combined to re-create a filtered version of the original image 102 orfurther processed as separate components.

A network of switches 108 selects one of three filters 110, 112, 114 forthe brightness component 104 of the image 102. The system 100 mayinclude any number of brightness-component filters, however, including asingle filter, and the current invention is not limited to anyparticular number or type of filter. In one embodiment, a low-passaveraging filter 110 may be selected by the switches 108 if the sourceimage 102 is simple, if only a small degree of filtering is required,and/or if system resources are limited. The low-pass averaging filter110 attenuates high-frequency signals in the brightness component 104,while allowing low-frequency signals to pass. In one embodiment, thelow-pass averaging filter 110 performs a blur function on the brightnesscomponent 104.

A median filter 112 may be used to filter the brightness component 104for images of medium complexity, if a medium amount of filtering isdesired, and/or if an average amount of system resources is available.As one of skill in the art will understand, the median filter 112processes the brightness component 104 pixel by pixel and replaces eachpixel with the median of it and surrounding pixels. For example, themedian filter 112 may consider a 3×3 window of pixels surrounding apixel of interest (i.e., nine total pixels). The median filter 112 sortsthe nine pixels by their brightness values, selects the value in themiddle (i.e., fifth) position, and replaces the pixel of interest withthe selected value. In one embodiment, the filter 112 is a rank orrank-median filter, and may select a pixel in any position in the sortedlist of pixels (e.g., the third or sixth position). In one embodiment,if the absolute difference between the selected value and the originalvalue is larger than the threshold, the original value is kept; if thedifference is smaller than or equal to the threshold, the ranked valueis assigned.

An adaptive filter 114 may be used to filter the brightness component104 for images of high complexity, if a large amount of filtering isdesired, and/or if a large amount of system resources is available. Theadaptive filter 114 selects a filtering technique based on thedynamically determined characteristics of the brightness component 104,as explained in greater detail below.

A low-pass averaging filter 116 (e.g., a 5×5 low-pass averaging filter)may be used to filter the color component 106. In one embodiment, thecolor component 106 is less complex than the brightness component and/oris less affected by low-light noise and thus requires less filtering.The filter 116 may be a temporal-averaging filter withsum-of-absolute-differences or any other type of similar filter. Thesystem 100 may include more than one color-component filter 116, and oneof the plurality of color-component filters 116 may be selected based onthe complexity of the color component 106, the availability of systemresources, and/or a desired level of filtering quality.

FIG. 2 illustrates a flowchart 200 for removing noise from a low-lightimage. A first filter is applied to a luma component of a low-lightimage (Step 202) and a second filter is applied to a chroma component ofthe low-light image (Step 204). The filtered luma component is combinedwith the filtered chroma component to produce a filtered low-light image(Step 206). The first filter may be the low-pass averaging filter 110,median/rank-median filter 112, or the edge/Gaussian-distribution-basedadaptive filter 114, as described above, and the second filter may bethe low-pass or temporal-averaging filter 116.

FIG. 3 is an illustration of one implementation 300 of the adaptivefilter 114. An edge-difference filter 302 detects edges in a lumacomponent 104 of an image 102. The edge-difference filter 302 may alsobe known as a difference filter. The edge-difference filter 302 maydetect edges in the luma component 104 while retaining high-frequencydetails therein. The edge-detection process divides the pixels in theluma component into edge and non-edge pixels.

A dilation-based filter 304 modifies the output of the edge-differencefilter 302 by distributing the results of the edge detection toneighboring pixels. The dilation-based filter may be modified to easeimplementation on, for example, embedded and/or DSP platforms. Forexample, if four pixels in a row are dilated, the four pixels may beshifted, depending on the pixel location, to align with a word boundary.In various embodiments, the dilation-based filter 304 is a morphologyfilter, a 3×4 dilation filter, or a 4×3 dilation filter. Thedilation-based filter 304 may expand, or dilate, regions of pixelsdesignated as edge pixels to incorporate other, nearby pixels. Forexample, a pixel having an intensity different from its neighbors may bethe result of low-light noise; but, if the location of the pixel is neara detected edge, the pixel may instead be the result of a real physicalfeature of the captured image. The dilation-based filter 304, bycorrelating such pixels occurring near detected edges to edge pixels,prevents their erroneous designation as noise-produced pixels.

Each non-edge pixel in the dilated luma component 104 is then analyzedagainst a neighboring region of pixels (e.g., a neighboring 3×3 block ofpixels). Depending on the differences between the analyzed pixel and itsneighbors, as computed by a Gaussian distribution engine 306, the pixelis assigned a new value according to assignment units 308-312 and outputby an output unit 314.

In greater detail, the Gaussian distribution engine 306 computes a meanand a variance of the Gaussian distribution of the block or windowsurrounding the analyzed pixel. The deviation of the pixel from the meanof the block is computed and compared with the variance. If thedifference between the pixel and the variance is much greater than themean (e.g., greater than three times the standard deviation), the pixelis likely the result of low-light noise. In this case, the median block308 replaces the pixel with the median of the block of pixels. If thedifference between the pixel and the variance is near the mean, thelow-pass filter 310 replaces the analyzed pixel with the result oflow-pass filtering the block of pixels. If the difference between thepixel and the variance is less than the mean, the pixel block 213 passesthe analyzed pixel to the output block 314 unchanged.

In general, the algorithm utilized by the assignment units 308-312 maybe generalized by the following equations:

If {(Analyzed Pixel)−(Mean of Block of Pixels)}>N×(Variance of Block ofPixels): Output=Median of Block of Pixels  (1)

If {(Analyzed Pixel)−(Mean of Block of Pixels)}>M×(Variance of Block ofPixels): Output=Result of Low-Pass Filter of Block of Pixels  (2)

If {(Analyzed Pixel)−(Mean of Block of Pixels)}>P×(Variance of Block ofPixels): Output=Original Analyzed Pixel  (3)

wherein P≦M≦N. That is, the output 314 is assigned the median 308 forlarge differences, the low-pass filter 310 for medium differences, andthe original pixel 312 for small differences. In one embodiment, theoperations performed by the above equations (1)-(3) are executed byspecially allocated hardware. In another embodiment, the medianoperation is performed by the median filter 112 and low-pass filteringis performed by the low-pass averaging filter 110, as shown in FIG. 1.

FIG. 4 illustrates an example luma component 400. An edge 402 isdetected between image regions 404 and 406. As described above, pixels408 near the edge 402 may be designated as edge pixels by thedilation-based filter 304. A first pixel 410 may be analyzed andcompared to its 3×3 surrounding pixels 412. In this case, because thedifference between the analyzed pixel 410 and the mean of the block ofpixels 412 is much greater (i.e., greater than a threshold N) than thevariance of the block of pixels 412 (i.e., there is a large discrepancybetween the luma value of the pixel 410 and its neighbors 412), thepixel 410 is replaced with the median of the 3×3 surrounding pixels 412.

In another example, another pixel 414 is analyzed and compared to itssurrounding pixels 416. Here, because the difference between theanalyzed pixel 414 and the mean of the block of pixels 412 is less thanthe first threshold N but greater than a second threshold M whencompared to the variance of the block of pixels 412, the pixel 414 isreplaced with the result of low-pass filtering the block 416. Finally,because the difference between a third analyzed pixel 418 and the meanof its surrounding block of pixels 420 is much less than a threshold Pwhen compared to the variance of the block of pixels 420, the pixel 418remains unchanged.

In one embodiment, the above-described system 300 analyzes every pixelin the luma component 104. In other embodiments, the system 300 analyzesonly a subset of the total pixels in the luma component 104. Forexample, the system 300 may analyze only even-numbered pixels (e.g.,every second pixel) in the luma component 104. The result of analyzingan even-numbered pixel may be applied not only to that pixel itself, butalso to a neighboring odd-numbered pixel (e.g., a pixel adjacent to theanalyzed even-numbered pixel in the same row). Because the two pixelsare neighbors, the result computed for one pixel is likely to be similarto the uncomputed result of the neighboring pixel, and applying theanalyzed pixel's result to both pixels may produce only a small error.Other subsets of pixels may be chosen for analysis, such as odd pixels,every Nth pixel, diagonal pixels, or rows/columns of pixels. Theanalyzed pixels may constitute 50% of the total pixels, as in theexample above, or any other percentage of total pixels.

FIG. 5 is a flowchart 500 illustrating a method for adaptively filteringnoise from a low-light image. An edge detected in the image is dilated(Step 502) using, e.g., the edge-difference filter 302 anddilation-based filter 304 described above. The edge-detection anddilation divides the image into edge and non-edge pixels, and pixels inthe non-edge region are compared to regions surrounding the pixels (Step504). Depending on the result of the comparison, as described above, thenon-edge pixels are optionally replaced (Step 506).

FIG. 6 is a block diagram 600 of a system for removing noise from alow-light image by dividing the image into sub-regions. A divisioncircuit 602 divides the image into two or more regions, and a filtercircuit 604 applies a first filter to luma components of each of theregions. Once each region has been filtered, a recombination circuit 606combines each filtered region to create a filtered image. In general,the regions may be any M×N size, for example, 16×16 pixels.

In one embodiment, the system 600 may be used to divide an image into anumber of regions that corresponds to a number of available filtercircuits 604. Each filter circuit 604 may include a system 100, asillustrated in FIG. 1, for removing low-light noise from each region.The filter circuit 604 may include a first filter for filtering a lumacomponent and a second filter for filtering a chroma component. Theplurality of regions may then be filtered simultaneously in parallel,thereby reducing the time required to filter the entire image. In otherembodiments, the number of regions is greater than the number of filtercircuits 604, and some regions are processed in parallel while othersare queued.

In another embodiment, only one filter circuit 604 is used to processeach image region in series. In this embodiment, the size of the imageregion may be defined by an amount of memory or other storage spaceavailable and/or the capabilities of the filter circuit 604. The size ofthe region may be adjusted to consume more or fewer resources, dependingon the constraints of a particular application. For example, anapplication having very limited memory may require a small region.History information for rows and columns of the regions or image may bestored and managed to ease data movement when switching and/or combiningimage regions.

FIG. 7 illustrates a method 700 for removing noise from a low-lightimage by dividing the image into sub-regions. The image is divided intoa plurality of regions (Step 702), and a first filter is applied (inseries or in parallel) to luma components of each of the regions (Step704). The separately filtered regions are combined into a filtered image(Step 706).

Applying the first filter may include low-pass filtering the region,median filtering the region, and/or adaptively filtering the region, asdescribed above with reference to FIG. 1. The adaptive filter compares apixel in the region to neighboring pixels and optionally replaces it. Asalso described above, a chroma component of the image may also be brokendown into image regions by the division circuit 602, filtered with asecond filter, and re-combined by the recombination circuit 606. Thesizes of the image regions of the chroma component may be the same as ordifferent from the sizes of the image regions of the luma component. Inone embodiment, the chroma component is processed as an entire image,due to its having less complexity, while the luma component is dividedand processed separately.

Embodiments of the present invention may be provided as hardware,software, and/or firmware. For example, the systems 100, 300, 600 may beimplemented on an embedded device, such as an ASIC, FPGA,microcontroller, or other similar device, and included in a video orstill camera. In other embodiments, elements of the systems 100, 300,600 may be implemented in software and included on a desktop, notebook,netbook, or handheld computer. In these embodiments, a webcam,cellular-phone camera, or other similar device may capture images orvideo, and the systems 100, 300, 600 may remove low-light noisetherefrom. The present invention may further be provided as one or morecomputer-readable programs embodied on or in one or more articles ofmanufacture. The article of manufacture may be any suitable hardwareapparatus, such as, for example, a floppy disk, a hard disk, a CD ROMdisk, DVD ROM disk, a Blu-Ray disk, a flash memory card, a PROM, a RAM,a ROM, or a magnetic tape. In general, the computer-readable programsmay be implemented in any programming language. Some examples oflanguages that may be used include C, C++, or JAVA. The softwareprograms may be further translated into machine language or virtualmachine instructions and stored in a program file in that form. Theprogram file may then be stored on or in one or more of the articles ofmanufacture.

Certain embodiments of the present invention were described above. Itis, however, expressly noted that the present invention is not limitedto those embodiments, but rather the intention is that additions andmodifications to what was expressly described herein are also includedwithin the scope of the invention. Moreover, it is to be understood thatthe features of the various embodiments described herein were notmutually exclusive and can exist in various combinations andpermutations, even if such combinations or permutations were not madeexpress herein, without departing from the spirit and scope of theinvention. In fact, variations, modifications, and other implementationsof what was described herein will occur to those of ordinary skill inthe art without departing from the spirit and the scope of theinvention. As such, the invention is not to be defined only by thepreceding illustrative description.

1. An adaptive filter for filtering noise from a low-light image, theadaptive filter comprising: a morphology filter for dilating an edgedetected in the image, thereby dividing the image into a non-edge regionand a dilated edge region; and a comparative filter for comparing apixel in the non-edge region to a region surrounding the pixel andoptionally replacing the pixel, based on a result of the comparison,with a new pixel derived at least in part from the region surroundingthe pixel.
 2. The adaptive filter of claim 1, further comprising adifference filter for detecting the edge in the image.
 3. The adaptivefilter of claim 1, wherein the morphology filter comprises a dilationfilter.
 4. The adaptive filter of claim 3, wherein the dilation filteris a 3×4 dilation filter.
 5. The adaptive filter of claim 1, wherein theregion surrounding the pixel corresponds to a 3×3 pixel region.
 6. Theadaptive filter of claim 1, wherein the adaptive filter compares adifference between the pixel and a mean of the region surrounding thepixel to a variance of the region surrounding the pixel.
 7. The adaptivefilter of claim 6, wherein the variance is a variance of a Gaussiandistribution of the region surrounding the pixel.
 8. The adaptive filterof claim 7, further comprising a circuit for computing the variance ofthe Gaussian distribution of the region surrounding the pixel.
 9. Theadaptive filter of claim 6, wherein the mean is a mean of a Gaussiandistribution of the region surrounding the pixel.
 10. The adaptivefilter of claim 7, further comprising a circuit for computing the meanof the Gaussian distribution of the region surrounding the pixel. 11.The adaptive filter of claim 1, wherein the comparative filter derivesthe new pixel based at least in part on a median value of the regionsurrounding the pixel.
 12. The adaptive filter of claim 1, wherein thecomparative filter derives the new pixel based at least in part on alow-pass filter value of the region surrounding the pixel.
 13. Theadaptive filter of claim 1, wherein the comparative filter replaces asecond pixel adjacent to the pixel with the new pixel.
 14. The adaptivefilter of claim 13, wherein the pixel is an even pixel and the secondpixel is an odd pixel.
 15. The adaptive filter of claim 1, wherein thecomparative filter preserves the pixel.
 16. A method for adaptivelyfiltering noise from a low-light image, the method comprising: dilatingan edge detected in an image to divide the image into a non-edge regionand a dilated edge region; comparing a pixel in the non-edge region to aregion surrounding the pixel; and optionally replacing the pixel, basedon a result of the comparison, with a new pixel derived at least in partfrom the region surrounding the pixel.
 17. The method of claim 16,further comprising detecting the edge in the image.
 18. The method ofclaim 16, wherein comparing the pixel to the region surrounding thepixel comprises comparing a difference between the pixel and a varianceof the region surrounding the pixel to a mean of the region surroundingthe pixel.
 19. The method of claim 18, wherein the variance is avariance of a Gaussian distribution of the region surrounding the pixel.20. The method of claim 18, wherein the mean is a mean of a Gaussiandistribution of the region surrounding the pixel.
 21. The method ofclaim 16, wherein the new pixel is derived based at least in part on amedian value of the region surrounding the pixel.
 22. The method ofclaim 16, wherein the new pixel is derived based at least in part on alow-pass filter value of the region surrounding the pixel.
 23. Themethod of claim 16, further comprising replacing a second pixel with thenew pixel, the second pixel being adjacent to the pixel.
 24. The methodof claim 23, wherein the pixel is an even pixel and the second pixel isan odd pixel.
 25. The method of claim 16, wherein the comparative filterpreserves the pixel.